Multi spectral imaging is used in many different applications and fields.
FIG. 1 shows a conventional DSLR (Digital Single-Lens Reflex) camera 1. The CCD (Charge-Coupled Device) and CMOS (Complementary Metal-Oxide-Semiconductor) sensors 3 of today's consumer grade DSLR and Mirrorless cameras are sensitive to a “full spectrum”, which is usually from about 170 nm to 1150 nm. This sensitivity of CCD and CMOS sensors exceeds human perception, which is limited to visible light of about 400 nm to 700 nm.
In some DSLR/Mirrorless cameras with an interchangeable lens mount 2, the sensors are sensitive to the spectrum of light beyond human perception. However, most camera manufacturers use a UVIR cut filter (not shown), which is installed in front of the sensor, to block infrared and ultraviolet light from “corrupting” the captured images, limiting the passage of the band of light in the visible (VIS) spectrum from about 400 nm to 700 nm (the spectrum of light visible to the human eye).
In some DSLR/Mirrorless cameras with an interchangeable lens mount, the UVIR cut filter can be removed from the camera, allowing the camera to operate as a “full spectrum” camera, covering wavelengths of light far exceeding the visible light. The most advanced CMOS sensors of today are sensitive to light beyond 1000 nm (NIR) and 200 nm (UV). The removal of the filter allows capturing of three different and distinctive bands of the spectrum using a single lens. One can install different multi bandpass filters allowing the capture of one of four combinations of three bands: NIR or red and green and blue or UV (NIR;G;B, NIR;G;UV, R;G;UV, R;G;B). The typical sensor of a consumer camera is based on a Bayer matrix, which means that the arrangement of color filters on the pixel array of an image sensor is of RGGB. Such an arrangement does not allow capturing more than three distinctive bands of the RGB spectrum, as there is no efficient way of separating more than a single band in the green wavelength.
Conventional multi spectral cameras are usually custom built, and their high cost presents a barrier for many. Still, multispectral cameras mounted in drones are used in agriculture to capture images that provide useful information to a farmer which helps him improve the yield of his crops. Such cameras are also used for medical analysis by dentists and dermatologists, and even in the world of cosmetics. The existing art varies, as there are a few types of technologies for multi spectral cameras:
1. Multi-cameras (each camera acquiring a narrow band wave of light);
2. Prism based camera (single lens, multi sensors);
3. Rotating filter camera (single focal length, time lapse between captured images of different wavelengths);
4. Multi-color sensor based RGBU (Red Green Blue UV) or any other color capturing by the sensor (instead of the typical RGGB (Red Green Green Blue) of a Bayer color filter array).
There are known quite a few simplified dual-band solutions, which include a Visual (400 nm to 700 nm wide) band and a Near Infra-Red band, using two synchronized cameras, but they cannot be considered as multi spectral. Conventional 4 bands solutions, whether based on a revolving filter, prism or multi lenses design, are of rather low quality, due to low resolution and poor color channels aligmnent. All the multispectral camera types detailed above are very expensive and usually of low resolution and inferior quality when compared to consumer grade DSLR or Mirrorless cameras. Existing multi-camera and multi-lens solutions also present great challenges regarding the alignment of the different color matrices and the focus of the captured images. When a rotating filter is used, for example, it is almost impossible to correct the chromatic aberration, as the lens is fixed, which makes it impossible to focus bands of light of Near Infrared (NIR) and UV through the same lens. Using a camera with such a rotating filter on an unmanned aerial vehicle (UAV) presents another challenge. Since the images are captured at different times and from different angles while the UAV is flying, a time lap of 33 ms between captured images could result in ˜1 m difference in the point of view of the camera, if the UAV is flying at 100 Kmph.
An effective multi spectral apparatus for agricultural analysis must include at least 3 narrow color bands (channels) of the spectrum of light: 550 nm; 650 nm or 740 nm; 850 nm or 950 nm and, preferably includes 5 or 6 color bands.
The sensors of today's consumer grade cameras, whether CMOS or CCD, are capable of capturing a rather broad spectrum of light, which is sufficient for many multi-spectral applications, such as NDVI (Normalized Difference Vegetation index) used in agriculture. However, the typical sensor of a consumer grade camera is based on a Bayer matrix sensor, which means that the arrangement of color filters on the pixel array of the image sensor is of RGGB. In order to use a Bayer filtering system based camera to capture 4, 5 or 6 channels, two cameras must inevitably be used. Combining two cameras for this kind of application is known, and was carried out and published by the USDA in “An Airborne Multispectral Imaging System Based on Two Consumer-Grade Cameras for Agricultural Remote Sensing” http://www.mdpi.com/2072-4292/6/6/5257. Such a combination is expensive and has quite a few disadvantages.
Some of the main challenges of multi spectral imaging are calibration and pixel alignment of the different color matrices, especially if these are acquired by different cameras. While the time of acquisition could vary by just few hundredths of a second, there could be a gap of tens of centimeters in the point of view. Analysis of data acquired through multi spectral applications, which make use of fast moving cameras, such as cameras borne by UAVs, use metrics like NDVI or NDRE, which typically subtract and/or divide values of the color matrices from one another ((NDVI=(NIR−VIS)/(NIR+VIS)). Thus, using multiple cameras in such applications is especially problematic, since differences between the focal points and inaccurate alignment can lead to false analysis of the data.
Alignment of the different color matrices is one of the most essential issues facing manufacturers of multi-spectral cameras. As stated above, NDVI and NDRE metrics and the like involve an analysis of the differences between color matrices. Hardware calibration of the cameras or lenses, which capture the different color matrices, is accurate to a certain point, but it can never be pixel accurate when dealing with high resolution cameras. When dealing with cameras with short effective lenses, it is even more challenging to align the color channels, which are acquired by the different cameras. Post-acquisition digital alignment of the matrices, based on the usually used metric of minimal difference, could be detrimental to the process, as digital matching (such as implementing global motion estimation techniques) is bound to align the different color matrices at the point of minimal difference between such matrices. Such techniques could actually minimize or even diminish the most important data, namely, the difference between the matrices.
Multi spectral imagery acquisition is also used for medical analysis by dentists and dermatologists, for skin analysis, dental inspection and more, and even in the world of cosmetics. However, current devices on the market for these implementations are also rather expensive and have two major shortcomings (in addition to their cost): Weight and User Accessibility. The current devices are expensive in consumer and small business terms, as they are custom built. An effective multi spectral apparatus for skin, hair and teeth analysis must include at least 3 narrow color bands (channels) of the spectrum of light: 365 nm; 550 nm; 650 nm. Preferably, it includes 5 or 6 color bands.
Removal of the UVIR cut filter is often used to modify the camera to an Infrared or Ultraviolet camera, for astronomic, medical or aesthetic purposes. Installing a multi bandpass filter on such a full spectrum camera may allow capture of three distinctive bands, as the sensor which is used in such cameras has, in most cases, a matrix of RGGB. This means that R, G, and B (red green blue) channels can be captured separately, but there is no way of capturing red and near infrared channels separately, as they are both captured by the same sensitive-to-red pixels of the sensor.
There are also solutions for capturing 3D images using two lenses with a single or dual sensor camera. Such a solution was developed by Zeiss almost a hundred years ago and a newer version of such a solution was developed also by Panasonic around 2010, for the consumer electronics market (for what seemed to be the rise of 3D TV). However, these solutions do not deal with the multispectral challenges of exposure balance, chromatic aberration and aligned focus, since they are based on identical lenses and identical spectral bandwidth, where such issues do not arise.
Mobile platforms like Smartphones, or similar universal processing boards designed for drones (such as Qualcomm's SnapDragon Flight) could be ideal in terms of pricing, weight and integration with a powerful processor, as the foundation for a multi spectral imagery apparatus. The sensors of today's Smartphones are capable of capturing a rather broad range of light, which is sufficient for many multi-spectral applications, such as NDVI (Normalized Difference Vegetation Index) used in agriculture. Some of these designs (like the SnapDragon Flight) include a single camera, and in recent years, vendors like LG Electronics even designed a 3D imagery capturing Smartphone, equipped with two cameras. However, there is no available multi spectral Smartphone, either for agricultural remote sensing, or for skin, hair and dental analysis. This is hardly surprising, as there are quite a few challenges which require an innovative approach in order to enable an efficient embodiment of such a light and compact apparatus.
However, the typical sensor of a consumer camera is based on a Bayer matrix, which means that the arrangement of color filters on the pixel array of an image sensor is of RGGB. In order to use a Bayer filtering system based camera for 4, 5 or 6 channels, the use of two cameras is inevitable. Combining two cameras for this kind of application is known, and was carried out and published b the USDA in “An Airborne Multispectral Imaging System Based on Two Consumer-Grade Cameras for Agricultural Remote Sensing” http://www.mdpi.com/2072-4292/6/6/5257. Such a combination is not practical for light UAVs and consumers due to its heavy weight and cost.
The very same challenges are relevant to skin analysis, where rather than VIS+NIR channels, VIS+Near UV are used for surface and under skin observation. The same challenges of the chromatic aberration and focus arise in this implementation
One additional challenge for a skin analyzing Smartphone or mobile device is the lens type. For skin analysis purposes, one of the multiple cameras should be able to capture a narrow band of (peak) 365 nm. Simple glass lenses do not allow the transmission of such a band and a different material, such as fused quartz or fused silica, must be used for the NUV lens.
Accordingly, there is a long felt need for a multispectral imaging apparatus suitable for use as a fast moving or UAV borne camera, or as a skin analysis device, and it would be very desirable if such a camera provided alignment of the different color matrices and focus of the captured images, all at a relatively low cost.